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The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )□ Responsive to communication(s) filed on . 

2a)D This action is FINAL. 2b)KI This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 1-20 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) ^ Claim(s) 1-20 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) ^ The specification is objected to by the Examiner. 

1 0) IEI The drawing(s) filed on 04 May 2001 is/are: a)D accepted or b)IEI objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1 .121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)D All b)D Some * c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. Q Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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Drawings 

Objections 

1 . The drawings are objected to because of the following. In Fig. 3, the Feature Extraction 1 2 should be 
labeled Feature Extraction Determinator 12, according to line 13-14 on page 6 of the Applicant's specification. Fig. 
2 fails to comply with 37 CFR 1.84(p)(5) because it does not include the following reference sign mentioned in the 
description: video conferencing system 1 (line 17, page 5 of Applicant's specification). A proposed drawing 
correction or corrected drawings are required in reply to the Office action to avoid abandonment of the application. 
The objection to the drawings will not be held in abeyance. 



Specification 

Objections 

2. The disclosure is objected to because of the following informalities. On page 4, line 13, the Applicant 
states, "the steps of identify at least one object with in an image". To correct the grammatical error, this excerpt 
should read, "the steps of identifying at least on object within an image". On page 6, line 22, the Applicant refers to 
Fig. 100. Clearly, the Applicant intended to refer to Fig. 1 . Correction should be made to reflect this. Appropriate 
correction is required. 

3. The use of the trademark Microsoft Windows 95 has been noted in this application (page 8, line 16 of 
Applicant's specification). It should be capitalized wherever it appears and be accompanied by the generic 
terminology. 
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4. Although the use of trademarks is permissible in patent applications, the proprietary nature of the marks 
should be respected and every effort made to prevent their use in any manner which might adversely affect their 
validity as trademarks. 



Claims 

Objections 

5. Claims 1, 8, and 14 are objected to because of the following informalities. These claims state, "identify at 
least one object with in an image". Clearly, the emphasized portion of the preceding excerpt is a grammatical and/or 
typographical error and this portion of claims 1, 8, and 14 should read, "identify at least one object within an 
image". Appropriate correction is required. 

Rejections Under 35 U.S.C. $ 102(1)) 

6. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the 
rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on sale in this country 
more than one year prior to the date of application for patent in the United States. 

7. Claims 1-20 are rejected under 35 U.S.C. 102(b) as being anticipated by Escher and Thalmann 
("Automatic 3D Cloning and Real-Time Animation of the Human Face", IEEE 1997). 

8. The following is in regard to Claim L Escher and Thalmann disclose a technique for automatically 
modifying a generic 3-dimensional (3D) model of the human face, according to a facial image depicted in a 
sequence of images. The disclosed technique is directed primarily toward video conferencing or other virtual 
collaborative environments (i.e. model-based communication systems). See Abstract of Escher and Thalmann. This 
method comprises the steps of: 
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(1 .a.) Identify at least one object (i.e. various facial features defined by feature points) within 
an image (e.g. Escher and Thalmann page 59, right column, paragraph 1 and paragraph 2, 
sentence 2 and page 63, left column, paragraph 2, sentences 1-2). 

(l.b.) Extracting feature position information (e.g. feature points or characteristic points) of the 
object(s). See, for example, Escher and Thalmann page 59, right column, paragraph 2, 
sentence 2 and the section entitled Characteristic Points. It should be apparent that these 
feature points encompass position information (Escher and Thalmann page 59, right 
column, paragraph 2, sentences 4-6). 

( 1 x.) Determining whether an adapted model is available based upon the extracted feature 
information. Refer to Real time animation - Image/Video Driven on page 63 of Escher 
and Thalmann and page 59, right column, paragraph 3. The determination as to whether 
an adapted model is obtainable (available) occurs during the feature extraction and 
tracking, upon the return of an array of MP A (minimal perceptible actions - Escher and 
Thalmann page 59, right column, paragraph 3). These MP As are used to deform the 
generic model according to the depicted facial expression and, therefore, define the 
adapted model. 

( 1 .d.) If available, using the adapted model in the model-based communication system. This 
step follows from the discussion above relating to (l.c). 
It has thus been shown that the method of Escher and Thalmann is a method for a model-based communication 
system comprises, at least, all elements claimed in Applicant's claim 1. Therefore, the teachings of Escher and 
Thalmann anticipate the method set forth in Applicant's claim 1. 

9. The following is in regard to Claim 2. As shown above, Escher and Thalmann disclose a method that is in 
accordance with claim 1. As alluded to above, the method of Escher and Thalmann operates by deforming a generic 
model based on facial expressions (defined by the aforementioned array of MP As) observed in an image or image 
sequence. Also refer to Escher and Thalmann page 63, left column, paragraphs 6-8, Figures 1-2, 4 and 7. In this 
way, the method of Escher and Thalmann further includes the step of acquiring a customized model related to the 
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object by modifying a generic model of the object. Therefore, the teachings of Escher and Thalmann anticipate the 
method set forth in Applicant's claim 2. 

10. The following is in regard to Claim 3. As shown above, Escher and Thalmann disclose a method that is in 
accordance with claim 2. As mentioned above and in the Abstract of Escher and Thalmann, the models, adapted to 
the video (image sequence) input, are used in a model-based communication system. As would be apparent to one of 
ordinary skill in the art, computer modeling systems or 3D animation systems, such as that of Escher and Thalmann, 
typically store models (in RAM or disk) prior to display. Taking this into account, the method of Escher and 
Thalmann thus includes the step of using and storing the customized model in the model-based communication 
system. Therefore, the teachings of Escher and Thalmann anticipate the method set forth in Applicant's claim 3. 

1 1 . The following is in regard to Claim 4. As shown above, Escher and Thalmann disclose a method that is in 
accordance with claim 3. The method of Escher and Thalmann further comprise an initialization phase (step). See 
Escher and Thalmann page 59, right column, paragraph 2 and Figure 1 (in particular, the gray box labeled Face 
Initialization). The initialization phase involves initializing (essentially fitting the model and model texture to the 
corresponding properties of the image(s) of the face) the generic model. When the analysis of the video input (e.g. 
feature extraction and tracking) reveals the facial expression contained therein, the Face Animator uses the derived 
MPA array and the generic model to generate the customized model (Escher and Thalmann Figure. 1 and page 59, 
right column paragraph 3). The customized model thus becomes available and used in the animation (Escher and 
Thalmann Figure. 1). In this way, the said initialization phase and subsequent analysis and animation of Escher and 
Thalmann's method, collectively, provide a step in accordance with claim 4. Therefore, the teachings of Escher and 
Thalmann anticipate the method set forth in Applicant's claim 4. 

12. The following is in regard to Claim 5. As shown above, Escher and Thalmann disclose a method that is in 
accordance with claim 1. As mentioned above, with regard to claim 1, Escher and Thalmann direct their teachings 
particularly toward model-based communication systems such as video conferencing or other virtual collaborative 
environments. See the Abstract of Escher and Thalmann. Therefore, the method of Escher and Thalmann conforms 
to the limitations set forth in Applicant's claim 5. 

13. The following is in regard to Claim 6. As shown above, Escher and Thalmann disclose a method that is in 
accordance with claim 1. It should be clear that Escher and Thalmann's method deals strictly with human face 
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models. In this way, the method of Escher and Thalmann conforms to the limitations set forth in Applicant's claim 
6. 

14. The following is in regard to Claim 7. As shown above, Escher and Thalmann disclose a method that is in 
accordance with claim 1. It should be clear that Escher and Thalmann's method constructs 3D models of the human 
face. Therefore, the method of Escher and Thalmann conforms to the limitations set forth in Applicant's claim 6. 

15. The following is in regard to Claims 8-13. These claims recite substantially the same limitations as claims 
1-6, respectively. Therefore, with regard to claims 8-13, remarks analogous to those presented above with regard to 
claims 1-6 are, respectively, applicable. 

16. The following is in regard to Claims 14-20. These claims recite substantially the same limitations as claims 
1- 7, respectively. Therefore, with regard to claims 14-20, remarks analogous to those presented above with regard 
to claims 1- 7 are, respectively, applicable. 



Citation of Relevant Prior Art 

17. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. 

1 8. The methods, and the various embodiments of those methods, proposed by the Applicant in claims 1-20 
generally belong to the field of Image-Based Modeling and/or Rendering. A wide range of literature was available at 
the time of the applicant's claimed invention that address most aspects of the Applicant's claimed subject matter. 

[1] Modeling and Rendering Architecture from Photographs, Ph.D. Thesis. 1996. Paul Debevec 
This paper is well-known and often cited in the field of image-based modeling Debevec uses 
an image based approach to generating 3D models of architectural objects. Taken broadly, 
Debevec's teachings can be shown to address steps (l.a)-(l.d), discussed above relative to 
claim 1 . 

[2] Analysis and Synthesis of Facial Image Sequences in Model-Based Image Coding . IEEE 
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Transactions on Circuits and Systems for Video Technology, June 1994. Choi, et al. 
Choi et al. disclose a method for analyzing sequences of facial images and, based on that 
analysis, synthesizing and modifying a 3-dimensional facial model corresponding to various 
facial expressions present in the facial images. This method is applied to face-to-face 
communication systems such as a video phone of teleconferencing system. In a manner similar 
to Escher and Thalmann, Choi et al. express facial expressions in terms of a "fixed" set of 
basis expressions called action units (AU). The teachings of Choi et al. can be shown to 
anticipate each element of the Applicant's claimed subject matter. In particular, the method of 
Choi et al. addresses the limitations of claim 4. To find a vector in action unit (AU) space that 
defines a model that accurately represents the input facial image, Choi et al. iteratively 
evaluate the error between the model-based image (a projection of the 3D model onto the 2D 
plane of the input image) and the actual input facial image. In doing so, the generic model is 
initialized and, on each iteration, deformed by the derived AU vector until an accurate 
adapted/cwjfo/mzed/deformed model is obtained (is available). This directly meets the 
limitations of claim 4. 

[3] Face to Virtual Face. IEEE, 1998. Thalmann, et al. 

Thalmann et al. provide an overview of a more comprehensive and sophisticated system that 
encompasses the methods of Escher and Thalmann, discussed above. 

[4] Modeling Facial Communication Between an Animator and a Synthetic Actor in Real Time. 
Proceedings of Modeling in Computer Graphics, 1993. 

[5] SMILE: A Multilayered Facial Animation System, IFIP, 1991. 

[6] Simulation of Facial Muscle Actions Based on Rational Free Form Deformations. 
Eurographics 1992. Kalra, et al. 

References [4]-[6] further elucidate the feature extraction and tracking and facial modeling of 
Escher and Thalmann. In particular, [5]-[6] provide a thorough dissertation of minimal 
perceptible actions (MP As) and how they may be used in modeling expressions. 
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[7] Synthesizing Realistic Facial Expression From Photographs. SIGGRAPH, 1998. Pighin, et al. 

Pighin et al. disclose a method that can be shown to include steps ( 1 .a)-( 1 .d) discussed above. 

This reference is also relevant to claims 2, 4, and 6-7. 
[8] International Patent Application Publication WO 98/01830: Image Processing. January 1998. 

Welsh, et al. 

Welsh et al. disclose a method of coding an image of a face by deforming a generic shape 
model and underlying muscle model by applying muscle parameters derived from the input 
image. Welsh et al. use the method of coding in a video-conferencing application. Welsh et al. 
address most of the Applicant's claimed subject matter. 



Any inquiry concerning this communication or earlier communications from the examiner should be 
directed to Kevin Siangchin whose telephone number is (703)305-7569. The examiner can normally be reached on 
9:00am - 5:30pm, Monday - Friday. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Amelia Au can 
be reached on (703)308-6604. The fax phone number for the organization where this application or proceeding is 
assigned is 703-872-9306. 

Information regarding the status of an application maybe obtained from the Patent Application Information 
Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR 
or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more 
information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the 
Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



ks - 04/22/04 



Examiner 
Art Unit 2623 



Kevin Siangchin 





